Specifying and Validating the Agent Performance Evaluation Methodology: the Symbiosis Use Case

Despite the plethora of frameworks and tools for developing agent systems, there is a remarkable lack of generalized methodologies for assessing their performance, while adequately addressing the unpredictable and complex nature of intelligent agents. In this paper, we present a generic methodology for evaluating agent performance, the Agent Performance Evaluation (APE) methodology that consists of representation tools, guidelines and techniques for organizing and using metrics, measurements and aggregated characterizations of performance. The main element of APE is the Metrics Representation Tree, a generic structure that enables efficient manipulation of evaluation-specific information. A formal specification of the proposed methodology is provided and its applicability is demonstrated through Symbiosis, an existing multi-agent system to be used as a testbed.